System and method for displaying real or virtual scene

ABSTRACT

A system and a method for displaying a scene are provided. The system includes a display configured to emit light, a spatial light modulator configured to modulate input light based on a transparency value, and at least one processor configured to acquire adjustment information including transparency of the spatial light modulator and light intensity information of the display from a plurality of pieces of view information corresponding to the scene and adjust an intensity value of the light emitted from the display and the transparency value of the spatial light modulator based on the adjustment information, wherein the plurality of pieces of view information are optical information of the scene, the optical information having been acquired at a plurality of viewpoints.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. §119(a) of a Russian patent application number 2017129073, filed on Aug.15, 2017, in the Russian Intellectual Patent Office, and of a Koreanpatent application number 10-2018-0077317, filed on Jul. 3, 2018, in theKorean Intellectual Property Office, the disclosure of each of which isincorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to an imaging technology. More particularly, thedisclosure relates to a system and a method for displaying a real orvirtual scene capable of generating high image quality three-dimensional(3D) images while addressing a vergence-accommodation conflict.

The disclosure enables a user to be immersed in a virtual reality (VR)of various tasks, such as 3D modeling, navigation, design, andentertainment. The disclosure may be employed in various head-mounteddevices (H IDs), such as VR glasses or helmets, which are beingincreasingly used in game and education industries at the moment.

2. Description of Related Art

This section is not provided to describe the technical features of thedisclosure, and thus the technical features of the disclosure are notlimited by this section. This section is to provide the outline of therelated art, which belongs to the same technical field as thedisclosure, to those of ordinary skill in the art and to thereby makeclear the technical importance due to differences between the relatedart and the disclosure.

Recently, VR technology has been increasingly used in various fields oflife within human society (traditional and well-known applications ingame and education industries). To popularize the VR technology andprovide for its long-term application, it is necessary to provide avisually comfortable interaction between users and reality.

Modern VR displays support various cues of human vision, for example,motion parallax, binocular disparity, binocular occlusion, and vergence.However, an accommodation cues of a human eye for virtual objects is notsupported by these displays. This causes a phenomenon calledvergence-accommodation conflict to occur. The vergence-accommodationconflict occurs because a human vision system needs to maintain acertain focal distance of eyeball lenses when viewing a 3D image, inorder to focus on an image formed and viewed by a display or a lens,while simultaneously a user has to change focal distances of the eyeballlenses based on distances to a virtual object according to the currentmovement of his or her eyes. In other words, the vergence-accommodationconflict occurs since virtual objects are viewed as if the virtualobjects were located at different “distances”, but the virtual objectsactually exist on a flat surface of a display screen abreast of eachother. This conflict between a virtual sequence and reality causesvisual discomfort, eye fatigue, eye tension, and headache.

At the moment, light field display technology aiming at addressing theissues of negative effects by delivering the same light as normallyreceived by eyes to the eyes under similar conditions to those of a reallife has been being developed.

An embodiment of such a display is disclosed in US 2014/0063077. In moredetail, this document discloses a display apparatus including one ormore light attenuation layers of which addresses are spatiallyassignable, and a controller configured to perform computations neededto control the display apparatus, and to address an optimization issueby using weighted nonnegative tensor factorization (NTF) formemory-efficient representation of a light field at a low density. ThisNTF requires high costs. Furthermore, known apparatuses have no mobilityand cannot be head-mounted.

An embodiment of the disclosure is disclosed in paper “The Light-FieldStereoscope: Immersive Computer Graphics via Factored Near-Eye LightField Displays with Focus Cues (ACM SIGGRAPH, Transactions on Graphics33, 5, 2015)” by F. Huang, K. Chen, and G. Wetzstein. This paperdiscloses a portable VR display supporting an initial high resolutionimage and the possibility of focusing a user's eyes on a virtual object,that is, the possibility of addressing the vergence-accommodationconflict. A light field appears on each eye, and a more natural visualexperience than that in existing near-eye displays is provided throughthe light field. The proposed display uses rank-1 light fieldfactorization. To implement the display described above, an expensivetime-division multi-image display or eye tracking unit is not required.However, the authors of the paper used computationally complicatednon-negative matrix factorization (NMF) for a solution.

Therefore, a need exists for a display system, e.g., a head-mountabledisplay suitable for a VR application, capable of addressing thevergence-accommodation conflict while generating a high-quality image.

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

SUMMARY

Aspects of the disclosure are to address at least the above-mentionedproblems and/or disadvantages and to provide at least the advantagesdescribed below. Accordingly, an aspect of the disclosure is to providean apparatus and a method for displaying a real or virtual scene withoutrequiring complex computation while addressing a vergence-accommodationconflict.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

In accordance with an aspect of the disclosure, a system for displayingan image in a unit of a scene is provided. The system includes a displayconfigured to emit light, a spatial light modulator configured tomodulate input light based on a transparency value, and at least oneprocessor configured to acquire adjustment information includingtransparency of the spatial light modulator and light intensity of thedisplay from a plurality of pieces of view information corresponding tothe scene and adjust an intensity value of the light emitted from thedisplay and the transparency value of the spatial light modulator basedon the adjustment information, wherein the plurality of pieces of viewinformation are optical information of the scene, which has beenacquired at a plurality of viewpoints.

In accordance with another aspect of the disclosure, a scene displaymethod of displaying an image in a unit of a scene is provided. Themethod includes receiving a plurality of pieces of view informationcorresponding to the scene, acquiring, from the plurality of pieces ofview information, adjustment information including light intensity oflight emitted from a display and transparency of a spatial lightmodulator configured to modulate the light, and adjusting an intensityvalue of the light emitted from the display and a transparency value ofthe spatial light modulator, based on the adjustment information,wherein the plurality of pieces of view information are opticalinformation of the scene, which has been acquired at a plurality ofviewpoints.

In accordance with another aspect of the disclosure, at least onenon-transitory computer-readable recording medium has recorded thereon acomputer-readable program for performing the method described above.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 illustrates a light field diagram according to a view array of aspecific scene captured at different viewpoints by using a camera arrayaccording to an embodiment of the disclosure;

FIG. 2 illustrates an extended view of a display system for displaying areal or virtual scene according to an embodiment of the disclosure;

FIGS. 3A and 3B illustrate spatial light modulators according to displaytypes of a mobile electronic device according to various embodiments ofthe disclosure;

FIG. 4 illustrates a display system including a belt for mounting to ahead according to an embodiment of the disclosure;

FIG. 5 is a flowchart of a method of operating a display systemaccording to an embodiment of the disclosure;

FIG. 6 illustrates a two-parameter light field expression by Levoy andHanrahan according to an embodiment of the disclosure;

FIG. 7 illustrates a weighted-matrix calculation method performed basedon geometric parameters of a system according to an embodiment of thedisclosure;

FIG. 8 illustrates a matrix consisting of views using a barycentriccoordinate system according to an embodiment of the disclosure;

FIG. 9 is a block diagram of a display system according to an embodimentof the disclosure;

FIG. 10 is a flowchart of a method of displaying a scene according to anembodiment of the disclosure;

FIG. 11 is a flowchart of a method of displaying a scene according to anembodiment of the disclosure;

FIG. 12 is a flowchart of a method of displaying a scene according to anembodiment of the disclosure;

FIG. 13 is a flowchart of an enhancing processing operation according toan embodiment of the disclosure;

FIG. 14 is a flowchart of a method of displaying a scene according to anembodiment of the disclosure; and

FIG. 15 is a flowchart of a method of displaying a scene according to anembodiment of the disclosure.

Throughout the drawings, like reference numerals will be understood torefer to like parts, components, and structures.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill will recognize that various changesand modifications of the various embodiments described herein can bemade without departing from the scope and spirit of the disclosure. Inaddition, descriptions of well-known functions and constructions may beomitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but, are merely used by theinventor to enable a clear and consistent understanding of thedisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of thedisclosure is provided for illustration purpose only and not for thepurpose of limiting the disclosure as defined by the appended claims andtheir equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

By the term “substantially” it is meant that the recited characteristic,parameter, or value need not be achieved exactly, but that deviations orvariations, including for example, tolerances, measurement error,measurement accuracy limitations and other factors known to those ofskill in the art, may occur in amounts that do not preclude the effectthe characteristic was intended to provide.

The term “various embodiments” used in the specification indicates thatthe term is used “illustratively or for description”. It is not analyzedthat an embodiment disclosed as “various embodiments” in thespecification is necessarily more preferred than the other embodiments.

Reference will now be made to embodiments, examples of which areillustrated in the accompanying drawings, wherein like referencenumerals refer to like elements throughout. In this regard, theembodiments may have different forms and should not be construed asbeing limited to the descriptions set forth herein. Accordingly, theembodiments are merely described below, by referring to the figures, toexplain aspects. As used herein, the term “and/or” includes any and allcombinations of one or more of the associated listed items.

FIG. 1 illustrates a light field diagram according to a view array of aspecific scene captured at different viewpoints by using a camera arrayaccording to an embodiment of the disclosure.

Referring to FIG. 1, in the specification, “light field” indicates avector function indicating an amount of light moving in an arbitrarydirection passing through an arbitrary point in a space. For example,“light field” indicates a spatial distribution of light fluxes comingout from a visualized image or scene. “Light field” is specified by aconversion direction and a specific value of radiant energy at eachpoint. A light field of a specific (real or virtual) scene may beapproximated by an array of a plurality of different views for acorresponding scene. The views may be respectively obtained fromdifferent viewpoints by using, for example, an array of cameras or microlenses of a plenoptic camera. Therefore, as shown in FIG. 1, views maybe slightly shifted with respect to each other.

FIG. 2 illustrates an extended view of a display system for displaying areal or virtual scene according to an embodiment of the disclosure.

Referring to FIG. 2, a display system 1 may include a mobile electronicdevice 2, a spatial light modulator 3, and an optical lens 4.

The embodiment shows a case where the mobile electronic device 2 is amobile or cellular phone, but those of ordinary skill in the art mayreplace the mobile or cellular phone by using devices capable ofimplementing the same functions, such as a laptop computer, a tabletcomputer, and a portable digital player. In addition, a dice image shownas an initial scene in FIG. 2 is not to limit the embodiment of thedisclosure, and the technical idea of the embodiment may be applied inthe same way to more complex images including objects and subjects invarious types and forms. According to an embodiment of the disclosure, adisplay of the mobile electronic device 2 may be an organic lightemitting diode (OLED) display or a display having a different pixelstructure.

The spatial light modulator 3 is disposed at the front of the display ofthe mobile electronic device 2 and may have a pixel structure having acontrollable color slide. The spatial light modulator 3 will bedescribed below.

FIGS. 3A and 3B illustrate spatial light modulators according to displaytypes of a mobile electronic device according to various embodiments ofthe disclosure.

Referring to FIG. 3A, a liquid crystal display 7 is used as the displayof the mobile electronic device 2, and as described in the document(Mukhin, I. A., Development of liquid-crystal monitors, BROADCASTINGTelevision and radiobroadcasting: 1 part—No. 2(46), March 2005, pp.55-56; 2 part—No. 4(48), June-July 2005, pp. 71-73), the liquid crystaldisplay 7 may include a backlighting unit, one pair of a firstpolarizing plate P1 and a second polarizing plate P2, and a first liquidcrystal layer LC1 located between the first polarizing plate P1 and thesecond polarizing plate P2. Herein, a second crystal layer LC2 and athird polarizing plate P3 located in the proximity of a user are used asthe spatial light modulator 3. Accordingly, compared with a method ofusing the first polarizing plate P1 located between the display and thefirst liquid crystal layer LC1, the first liquid crystal layer LC1, andthe second polarizing plate P2 next to the first liquid crystal layerLC1 as a spatial light modulator, in a method of using the secondcrystal layer LC2 and the third polarizing plate P3 as the spatial lightmodulator 3, the number of polarizing plates used for a spatial lightmodulator may be reduced, and thus a size of the display system 1 (notshown in FIG. 3A) may be reduced.

Referring to FIG. 3B, an OLED display 8 may be used as the display ofthe mobile electronic device 2. As described with reference to FIG. 3A,a fourth polarizing plate P4, a liquid crystal layer LC, and a fifthpolarizing plate P5 may be used as the spatial light modulator 3.

As described with reference to FIG. 2, the optical lens 4 is located atthe rear of the spatial light modulator 3 at a viewpoint of the user ofthe display system 1 and is also located at the front of one eye of theuser. Optical lenses having the same form as the optical lens 4 may bearranged at the front of the other eye of the user. A set of theselenses constitutes an optical lens device.

A transparency value of pixels of the spatial light modulator 3 and anintensity value of pixels of the display of the mobile electronic device2 may be variably changed by control signals provided from at least oneprocessor or controller (not shown) included in the display system 1. Anadjustment operation for the transparency and intensity will bedescribed when a method of operating the display system 1 is described.

FIG. 4 illustrates a display system including a belt for mounting to ahead according to an embodiment of the disclosure.

Referring to FIG. 4, the above-described components of the displaysystem 1, particularly, the mobile electronic device 2 and the spatiallight modulator 3 shown together in FIG. 2, may be accommodated in acase or enclosure 5 (see FIG. 4) made of a proper material, such asplastic or a synthetic material. In addition, to provide the possibilityof mounting the display system 1 to the head of the user, for example, aspecific mounting unit disposed on a leather belt 6 (see FIG. 4)connected to the case or enclosure 5 may be used. According to anembodiment of the disclosure, the case or enclosure 5 may be virtualreality (VR) glasses or a VR helmet.

FIG. 5 is a flowchart of a method of operating a display systemaccording to an embodiment of the disclosure.

Referring to FIG. 5, an operation of the display system 1 will bedescribed below. Particularly, operations performed by the processor orcontroller described above will be described with reference to FIG. 5.

In operation S1, the processor or controller receives a set of views ofa real or virtual scene, for example, dice shown in FIG. 2. Each view ofthe real or virtual scene is specified by a field of view defined for ascene, as described with reference to FIG. 1. A set of views of a scenemay be acquired by using a plenoptic camera, for example, Lytro Illum.The set of the acquired views may be stored in a memory of the mobileelectronic device 2. In this case, the processor or controller mayaccess the memory of the mobile electronic device 2 to extract a set ofviews of a scene for subsequent processing.

According to an embodiment of the disclosure, the processor orcontroller may form a set of views of a scene by itself by using arendering program.

In operation S1, the processor or controller may generate a matrix ofthe views by using geometric parameters of a system (for example, adistance between a display of a mobile electronic device and a spatiallight modulator, a focal distance of a lens, and distances from the lensto the display and the modulator in each view).

FIG. 6 illustrates a two-parameter light field expression by Levoy andHanrahan according to an embodiment of the disclosure.

Referring to FIG. 6, when the matrix of the views is generated, theprocessor or controller may be based on a two-parameter light fieldexpression by Levoy and Hanrahan. FIG. 6 shows an xy plane and a uvplane of a light field. Referring to FIG. 6, the light field may berepresented by a four-dimensional (4D) function L(x, y, u, v) indicatingthe intensity of light in an optical space, which is incident to onearbitrary dot on the xy plane after passing through one arbitrary dot onthe uv plane under the expression described above.

FIG. 7 illustrates a weighted-matrix calculation method based ongeometric parameters of a system according to an embodiment of thedisclosure.

Referring to FIG. 7, according to a simple geometric consideration withreference to FIG. 7, integer coordinates of points at which lightcrosses images on a display and a modulator, the integer coordinatesbeing derived from an angle of the display or the modulator, are shown,and virtual ghosts of the display and the modulator are calculated asbelow.

$\begin{matrix}{x_{k} = \left\lfloor {\frac{1}{M_{k}p_{k}}\left( {\frac{W}{2} + {x \pm {\frac{d_{k}}{d_{cn}}\left( {u - x} \right)}}} \right)} \right\rfloor} & {{Equation}\mspace{14mu} 1} \\{y_{k} = \left\lfloor {\frac{1}{M_{k}p_{k}}\left( {\frac{H}{2} + {y \pm {\frac{d_{k}}{d_{cn}}\left( {v - y} \right)}}} \right)} \right\rfloor} & {{Equation}\mspace{14mu} 2}\end{matrix}$

According to an embodiment of the disclosure, in Equations 1 and 2, kdenotes 1 or 2, wherein 1 and 2 correspond to the modulator and thedisplay, respectively. The signs + and − included in ± of Equations 1and 2 correspond to the modulator and the display, respectively, and M₁and M₂ denote magnification constants of the virtual ghosts of themodulator and the display, respectively. In addition, p₁ and p₂ denotepixel sizes of the modulator and the display, respectively, W and Hdenote a height and a length of a physical view image on the xy plane ofthe light field (d_(k) denoting a relative location or distance betweenthe xy plane of the light field and a virtual ghost is selected toacquire best image quality), and d_(cn) denotes a distance from aneye-lens plane to a light field plane.

In a light field factorization operation, the light field L(x, y, u, v)is factorized to a multiplication of transparency t(x₁, y₁) of thespatial light modulator and light intensity l(x₂, y₂) of the display.

L(x,y,u,v)≈t(x₁,y₁)l(x₂,y₂)   Equation 3

As described with respect to Equations described above, x₁, x₂, y₁, andy₂ may be represented as x, y, u, and v through Equation 1 and Equation2.

However, this kind of tensor factorization is complex, and thus thereexists high calculation burden. Therefore, the disclosure illustrates anembodiment of reducing this high calculation burden by using a simplermatrix factorization method than a tensor factorization method. Formatrix factorization, t and l denoting transparency and intensity may befactorized to vectors a and b as follows.

a _(i) =t(x ₁ , y ₁),   Equation 4

b _(j) =l(x ₂ , y ₂),   Equation 5

i=y ₁ w ₁ +x ₁,   Equation 6

j=y ₂ w ₂ +x ₂,   Equation 7

Herein, w_(k) denotes a width of images of the modulator or the displaycorresponding to a value of k and is measured based on the number ofpixels measured along an x axis.

A value of the light field L(x, y, u, v) is “encapsulated” to an elementT_(ij) of a matrix of views, and thus Equation may be replaced byEquation 8.

T_(ij)≈a_(i)b_(j)   Equation 8

In operation S3, the processor or controller generates an adjustmentmatrix indicating a product of a column vector indicating a transparencyvalue of pixels of the spatial light modulator and a row vectorindicating a brightness value of pixels of the display of the mobileelectronic device. Herein, elements of the column vector and the rowvector are selected such that the adjustment matrix is approximately thesame as the matrix of the views.

In more detail, an element (I, j) of the adjustment matrix is obtainedwhen light passes through a jth pixel of the display and an ith pixel ofthe spatial light modulator. When it is assumed that the matrix of theviews is T, and the transparency and the intensity described above are aand b, the fact that the matrix of the views is “approximately the same”as the adjustment matrix indicates that T≈ab^(T).

This optimization operation may be addressed by various methods.According to an embodiment of the disclosure, the optimization operationmay be performed by using weighted rank-1 residue iteration (WRRI). Adetailed operation of the WRRI is described in the related art (forexample, HO, N.-D., Nonnegative Matrix Factorization Algorithms andApplications, PhD thesis, Universit'e catholique de Louvain, 2008; andHEIDE et al., Cascaded displays: spatiotemporal superresolution usingoffset pixel layers, ACM Transactions on Graphics (TOG)—Proceedings ofACM SIGGRAPH 2014, Volume 33, Issue 4, July 2014).

The number of views is limited, and thus minimization is requiredthrough limitation of the number of elements to be used for acomputation. Accordingly, a weighted matrix W determined such thatT≈Wab^(T) is provided. Herein, the weighted matrix W includes only aweighted constant for a part where views of a scene are “encapsulated”and has a value of zero for the remaining parts. The optimizationoperation continues until elements of the vectors a and b causing theadjustment matrix to be most approximate with the matrix of the viewsare found out. Equation 9 is an embodiment of the optimizationoperation.

$\begin{matrix}{\underset{{a_{i} \geq 0},{b_{i} \geq 0}}{argmin}\frac{1}{2}{{{W{^\circ}}\left( {T - {ab}^{T}} \right)}}_{2}^{2}} & {{Equation}\mspace{14mu} 9}\end{matrix}$

In Equation 9, symbol ∥ ∥₂ denotes L₂-norm satisfying

${{A}_{2}^{2} = {\sum\limits_{ij}^{\;}A_{ij}^{2}}},$

and an operation symbol ° denotes a product between elements, forexample, an Hadamard product, performed for element of the vectors a andb until the adjustment matrix is approximately the same as the matrix ofthe views.

The centers of the pixels of the display included in the mobileelectronic device 2 and the spatial light modulator 3 are matched witheach other, number “1” is assigned to elements of the matrix Wcorresponding to corresponding views (for example, i and j where Tij isencapsulated from the views), and the remaining elements of the matrixare filled with zero. If this matching does not occur, the matrices Tand W are constructed using barycentric coordinates, and distortion ofthe views of the scene in a subsequent processing operation is preventedthrough the construction using barycentric coordinates.

FIG. 8 illustrates a matrix consisting of views using a barycentriccoordinate system according to an embodiment of the disclosure.

Referring to FIG. 8, a construction operation using barycentriccoordinates is described. Referring to FIG. 8, λ and μ denotecoordinates (pixel centers) of a point marked in an X shape on the planeof the spatial light modulator 3, and w₀₀, w₀₁, w₁₀, and w₁₁ are valuesallocated to four elements specified by coordinates (x_(k), y_(k)),(x_(k)+1, y_(k)), (x_(k), y_(k)+1), and, (x_(k)+1, y_(k)+1). A sum ofw₀₀, w₀₁, w₁₀, and w₁₁ is 1, and thus a unit weight is allocated to fourneighboring elements. To construct the matrix of the views, each pixelvalue in each of four elements may be iterated four times.

According to an embodiment of the disclosure, an access of anothermethod is also possible. For example, when the matrix of the views isconstructed, values of the light field are allocated as respectiveweights to four elements specified by coordinates (x_(k), y_(k)),(x_(k)+1, y_(k)), (x_(k), y_(k)+1), and, (x_(k)+1, y_(k)+1) according tobarycentric coordinates. In this case, among the elements of theweighted matrix W, elements corresponding to non-zero elements of thematrix T have a value of 1, and the remaining elements have a value of0.

In operation S4, when components of the vectors a and b are identified,the processor or controller adjusts the intensity value 1 of the pixelsof the display of the mobile electronic device 2 according to thecomponents of the vector b and adjusts the transparency value t of thepixels of the spatial light modulator 3 according to the components ofthe vector a. Equations 4 and 5 described above mathematically representa relationship among a, b, t, and l. Through these operations, a lightfield of a scene, which is approximately the same as observed by a userin the real, for example, as if a 3D effect is provided when the userviews the scene, may be obtained.

According to an embodiment of the disclosure, the processor orcontroller may perform a pre-processing operation for each view of aprevious scene before proceeding to operations S2 to S3. Thepre-processing operation is an operation of enhancing details of viewsof a scene. In the pre-processing operation, a defined view (a detail tobe enhanced) of a scene is segmented to overlapping units includinggroups of pixels of the display of the mobile electronic device 2. Afollowing operation for each unit is performed.

First, a color of each pixel is converted into a YUV color model,wherein Y denotes a brightness component, and U and V denotecolor-difference components.

For each pixel, a separation operation for the brightness component Y isperformed. Next, the brightness component Y is added to a bright channelY for all pixels.

To obtain a Fourier spectrum, the bright channel is processed usingFourier transform. To smooth the spectrum at a boundary, a Gaussianwindow is used. The details are searched and enhanced using phasecongruency analysis in the Fourier spectrum. In addition, to obtain anew bright channel Y′, a Fourier inverse transform operation isperformed.

The phase congruency analysis is now described. As known, values of aFourier spectrum are complex numbers. The complex numbers are specifiedby an absolute value and an angle of deviation (that is, phase). Inother words, the complex numbers may be expressed in a form of 2D vectorhaving the same length and phase and the same direction as the absolutevalue. A search operation on a detail indicates an operation ofseparating vectors orienting one direction (together with specificdivergence), and an enhancing operation on the detail indicatesincreasing a length of retrieved vectors, that is, an operation ofincreasing a magnitude of the absolute value.

After performing the operations described above, all processed units arecombined such that overlapping is smoothly processed using a Gaussianwindow. Next, for each pixel, the new color model Y′ and the initialcomponents U and V are combined as a color model Y′UV. The color modelY′UV is converted into a color model RGB, and accordingly, a determinedview of a scene may be acquired as the color model RGB.

FIG. 9 is a block diagram of a display system according to an embodimentof the disclosure.

Referring to FIG. 9, a display system 900 may display a scene such thata light field which provides an experience approximate to a 3D effect inthe real to a user is provided, when a real or virtual scene isdisplayed.

The display system 900 may include a mobile electronic device 910 and aspatial light modulator 920. According to an embodiment of thedisclosure, the display system 900 may further include an optical lens(not shown). However, according to an embodiment of the disclosure, theoptical lens is not necessary required as a separated component. Theoptical lens may be replaced by a medium having the same opticalcharacteristics as the optical lens or included in the spatial lightmodulator 920.

The mobile electronic device 910 is a portable electronic device and maybe implemented in various forms, such as a smartphone, a tabletcomputer, a personal digital assistant (PDA), and a portable multimediaplayer (PMP).

The mobile electronic device 910 may include a processor 911 and adisplay 912. Although FIG. 9 shows that the processor 911 is included inthe mobile electronic device 910, this is not mandatory. According to anembodiment of the disclosure, the processor 911 may be located outsidethe mobile electronic device 910 and may control the mobile electronicdevice 910 and the spatial light modulator 920. For example, theprocessor 911 may be included in VR glasses or a VR helmet, which is acase in which the mobile electronic device 910 and the spatial lightmodulator 920 are accommodated.

The display 912 provides light to display a scene. According to anembodiment of the disclosure, the display 912 may include a liquidcrystal display mounted in the mobile electronic device 910. Forexample, the display 912 may include a backlight of the mobileelectronic device 910. In addition, the display 912 may include a liquidcrystal of the mobile electronic device 910.

The processor 911 may control the mobile electronic device 910 and thespatial light modulator 920 to perform a display operation of thedisplay system 900.

Although FIG. 9 shows that the spatial light modulator 920 is locatedoutside the mobile electronic device 910, this is not mandatory.According to an embodiment of the disclosure, the spatial lightmodulator 920 may be included in the mobile electronic device 910 andmodulate light provided from the display 912.

The processor 911 may receive a plurality of pieces of view informationwith respect to a scene to be displayed. According to an embodiment ofthe disclosure, the scene may be a virtual scene or a real scene. Theplurality of pieces of view information are optical information of ascene, which has been acquired at a plurality of viewpoints. Accordingto an embodiment of the disclosure, the plurality of pieces of viewinformation may be a set of a plurality of pieces of view informationacquired by photographing a real scene at a plurality of viewpoints. Forexample, the plurality of pieces of view information may be an intrinsicimage acquired from a plurality of matched cameras having differentviewpoints. According to an embodiment of the disclosure, the pluralityof pieces of view information may be a set of a plurality of pieces ofview information corresponding to a virtual scene formed using arendering program. According to an embodiment of the disclosure, theprocessor 911 may form a plurality of pieces of view informationcorresponding to a virtual scene by itself.

The processor 911 may acquire adjustment information from the pluralityof pieces of view information. The adjustment information may includeinformation regarding transparency of the spatial light modulator 920and light intensity of the display 912.

The light intensity of the display 912 indicates intensity of lightemitted by each pixel of the display 912. The light intensity of thedisplay 912 may be variably changed under control of the processor 911.The transparency of the spatial light modulator 920 indicates an opticalinfluence of each pixel of the spatial light modulator 920 to lighttransmitting through the spatial light modulator 920 and may includecolor transparency.

According to an embodiment of the disclosure, the adjustment informationmay include a view matrix that is a matrix including each viewinformation included in the plurality of pieces of view information,which is generated based on a geometric parameter. For example, theprocessor 911 may generate the view matrix from the plurality of piecesof view information. The view matrix is a matrix representing a lightfield of a corresponding scene to be displayed.

In addition, the processor 911 may perform light field factorization onthe generated view matrix. As described with reference to FIG. 7, alight field indicating light passing through a certain pixel of thedisplay 912 and a certain pixel of the spatial light modulator 920 maybe represented by a function of intensity of the display 912 andtransparency of the spatial light modulator 920. According to anembodiment of the disclosure, the processor 911 may factorize a givenview matrix to a product of a matrix indicating intensity of the display912 and a matrix indicating transparency of the spatial light modulator920, and this is called light field factorization.

According to an embodiment of the disclosure, the light fieldfactorization may be approximately achieved. Hereinafter, the lightfield factorization will be described below. In the embodiment below, itis described that the matrix indicating the intensity of the display 912is a row vector and the matrix indicating the transparency of thespatial light modulator 920 is a column vector, but this is onlyillustrative, and the technical features of the disclosure are notlimited thereto. The processor 911 may factorize a view matrix to aproduct of various types of matrices.

According to an embodiment of the disclosure, the processor 911 mayperform the light field factorization by using a WRRI algorithm. TheWRRI algorithm has a better processing speed and a less computationvolume than a non-negative matrix factorization (NMF) algorithm, andthus processor 911 may perform real-time processing at a higher speed byusing the WRRI algorithm than a speed using the NMF algorithm.

In more detail, as described with reference to Equation 9, the processor911 may calculate optimized intensity of the display 912 and optimizedtransparency of the spatial light modulator 920 through an Hadamardproduct with respect to a given light field by using the WRRI algorithm.According to an embodiment of the disclosure, the processor 911 may forma row vector indicating intensity of the display 912, a column vectorindicating transparency of the spatial light modulator 920, and anadjustment matrix indicating a product of the row vector and the columnvector. The processor 911 may select a row vector and a column vector byusing the WRRI algorithm such that an adjustment matrix is approximatelythe same as a view matrix.

The processor 911 may adjust an intensity value of light emitted fromthe display 912 and a transparency value of the spatial light modulator920, based on the adjustment information.

According to an embodiment of the disclosure, the processor 911 mayadjust intensity of the display 912 and transparency of the spatiallight modulator 920 based on a light field factorization result of aview matrix. For example, the processor 911 may form a row vector and acolumn vector forming an adjustment matrix which is approximately thesame as the view matrix and adjust the intensity of the display 912 andthe transparency of the spatial light modulator 920 based on the rowvector and the column vector. According to an embodiment of thedisclosure, intensity of each pixel of the display 912 may be adjustedin response to an intensity control signal provided from the processor911. In addition, transparency of each pixel of the spatial lightmodulator 920 may be adjusted in response to a transparency controlsignal provided from the processor 911.

The optical lens delivers, to the user, light which has passed throughthe display 912 and the spatial light modulator 920. The display system900 provides a light field which provides an experience approximate to a3D effect in the real to the user by providing light concentratedthrough the optical lens to the user.

FIG. 10 is a flowchart of a method of displaying a scene according to anembodiment of the disclosure.

Referring to FIG. 10, in operation S1010, a processor receives aplurality of pieces of view information corresponding to a scene.According to an embodiment of the disclosure, the scene may be a realscene or a virtual scene. The plurality of pieces of view informationmay be acquired by photographing the scene at different viewpoints. Forexample, the plurality of pieces of view information may have arelationship of being captured by being sequentially shifted at apreviously defined angle.

According to an embodiment of the disclosure, the plurality of pieces ofview information may be a set of a plurality of pieces of viewinformation acquired by photographing a real scene at a plurality ofviewpoints. For example, the plurality of pieces of view information maybe an intrinsic image acquired from a plurality of matched camerashaving different viewpoints. According to an embodiment of thedisclosure, the plurality of pieces of view information may be a set ofa plurality of pieces of view information corresponding to a virtualscene formed using a rendering program. According to an embodiment ofthe disclosure, the processor may form a plurality of pieces of viewinformation corresponding to a virtual scene by itself.

In operation S1020, the processor acquires adjustment information fromthe plurality of pieces of view information. The adjustment informationmay include information regarding transparency of a spatial lightmodulator and light intensity of a display. The light intensity of thedisplay indicates intensity of light emitted by each pixel of thedisplay. The transparency of the spatial light modulator indicates anoptical influence of each pixel of the spatial light modulator to lighttransmitting through the spatial light modulator and may include colortransparency. According to an embodiment of the disclosure, theadjustment information may include a view matrix.

In operation S1030, the processor may control the transparency of thespatial light modulator and a light intensity value of the display basedon the adjustment information. According to an embodiment of thedisclosure, intensity of each pixel of the display may be adjusted inresponse to an intensity control signal provided from the processor. Inaddition, transparency of each pixel of the spatial light modulator maybe adjusted in response to a transparency control signal provided fromthe processor.

Light which has passed through the display and the spatial lightmodulator may be delivered to a user through an optical lens. Thedisplay method according to the disclosure may provide a light fieldwhich provides an experience approximate to a 3D effect in the real to auser by providing light concentrated through the optical lens to theuser.

FIG. 11 is a flowchart of a method of displaying a scene according to anembodiment of the disclosure.

Referring to FIG. 11, in operation S1110, a processor receives aplurality of pieces of view information corresponding to a scene.According to an embodiment of the disclosure, the scene may be a realscene or a virtual scene. The plurality of pieces of view informationmay be acquired by photographing the scene at different viewpoints.

In operation S1120, the processor may acquire a view matrix included inadjustment information from the plurality of pieces of view information.For example, the processor may generate a view matrix that is a matrixincluding each view information included in the plurality of pieces ofview information, based on a geometric parameter. The view matrix is amatrix representing a light field of a corresponding scene to bedisplayed.

In operation S1130, the processor may factorize the given view matrix toa product of a matrix indicating light intensity of a display includedin a mobile electronic device and a matrix indicating transparency of aspatial light modulator.

According to an embodiment of the disclosure, the processor may performthe factorization by using a WRRI algorithm. In more detail, asdescribed with reference to Equation 9, the processor may calculateoptimized light intensity of the display and optimized transparency ofthe spatial light modulator through an Hadamard product with respect toa given light field by using the WRRI algorithm.

In operation S1140, the processor may control the transparency of thespatial light modulator and the light intensity of the display based ona result of the factorization. For example, the processor may factorizethe view matrix to a row vector and a column vector, control thetransparency of the spatial light modulator based on the column vector,and control the light intensity of the display based on the row vector.According to an embodiment of the disclosure, transparency of each pixelof the spatial light modulator may be adjusted in response to atransparency control signal provided from the processor, and intensityof each pixel of the display may be adjusted in response to an intensitycontrol signal provided from the processor.

A display system may display a scene to a user based on the operationsdescribed above. According to an embodiment of the disclosure, lightemitted from the display based on the adjusted light intensity isdelivered to the user through an optical lens by passing through thespatial light modulator having the adjusted transparency. The displaysystem may provide a light field which provides an experienceapproximate to a 3D effect in the real to the user by providing lightconcentrated through the optical lens to the user.

FIG. 12 is a flowchart of a method of displaying a scene according to anembodiment of the disclosure.

Referring to FIG. 12, a display system may perform an enhancingprocessing operation prior to acquiring adjustment information. Thedisplay system may provide a relatively clear and realistic experienceto a user by using the enhancing processing operation to enhance adetail of a view matrix.

In operation S1210, a processor receives a plurality of pieces of viewinformation corresponding to a scene. According to an embodiment of thedisclosure, the scene may be a real scene or a virtual scene. Theplurality of pieces of view information may be an intrinsic imageacquired from a plurality of matched cameras having differentviewpoints. Alternatively, the processor may form a plurality of piecesof view information corresponding to a virtual scene by itself by usinga rendering program.

In operation S1220, the processor performs the enhancing processingoperation on the pieces of view information. The enhancing processingoperation is an operation of enhancing only a detail while maintainingcolor information of the pieces of view information as it is. Accordingto an embodiment of the disclosure, for the enhancing processingoperation, the processor may separate only a brightness channel fromeach view information and perform a processing operation on thebrightness channel. According to an embodiment of the disclosure, theprocessor may use Fourier transform and phase congruency analysis forthe enhancing processing operation.

In operation S1230, the processor may generate adjustment information.According to an embodiment of the disclosure, the processor may generatea view matrix that is a matrix including each view information includedin the plurality of pieces of view information, based on a geometricparameter. The view matrix is a matrix representing a light field of acorresponding scene to be displayed.

According to an embodiment of the disclosure, the processor mayfactorize the enhancing-processed view matrix to a product of vectors.For example, the processor may factorize the enhancing-processed viewmatrix to a product of a matrix indicating light intensity of a displayincluded in a mobile electronic device and a matrix indicatingtransparency of a spatial light modulator.

According to an embodiment of the disclosure, the processor may performthe factorization by using a WRRI algorithm. In more detail, asdescribed with reference to Equation 9, the processor may calculateoptimized intensity of the display and optimized transparency of thespatial light modulator through an Hadamard product with respect to agiven light field by using the WRRI algorithm.

In operation S1240, the processor may control the transparency of thespatial light modulator and a light intensity value of the display basedon the adjustment information. For example, the processor may factorizethe view matrix to a column vector and a row vector, control thetransparency of the spatial light modulator based on the column vector,and control the light intensity of the display based on the row vector.According to an embodiment of the disclosure, transparency of each pixelof the spatial light modulator may be adjusted in response to atransparency control signal provided from the processor, and intensityof each pixel of the display may be adjusted in response to an intensitycontrol signal provided from the processor.

The display system may display a scene to the user based on theoperations described above. According to an embodiment of thedisclosure, light emitted from the display based on the adjusted lightintensity is delivered to the user through an optical lens by passingthrough the spatial light modulator having the adjusted transparency.The display system may provide a light field which provides anexperience approximate to a 3D effect in the real to the user byproviding light concentrated through the optical lens to the user.

FIG. 13 is a flowchart for describing the enhancing processing operationin more detail according to an embodiment of the disclosure.

Referring to FIG. 13, in operation S1221, a processor extracts abrightness channel of view information, on which the enhancingprocessing operation is to be performed.

According to an embodiment of the disclosure, the processor may segmentthe view information into a plurality of units which overlap each other,to extract the brightness channel. Each unit may include a pre-definedplurality of pixels.

According to an embodiment of the disclosure, the processor may converta color space model of the view information into a color space modelhaving a brightness channel to extract the brightness channel. Forexample, the processor may convert the color space model of the viewinformation into a YUV color space model or a YIQ color space model. Theembodiment illustrates that the color space model of the viewinformation is converted into the YUV color space model. However, thetechnical features of the disclosure are not limited to YUV color spaceinformation and may also be applied to other color spaces having abrightness channel.

In a YUV color space, a Y channel indicates information regardingbrightness, and U and V channels indicate information regarding colors.For example, the U channel is a value obtained by subtracting abrightness component from a blue (B) channel of an RGB color space, andthe V channel is a value obtained by subtracting the brightnesscomponent from a red (R) channel.

According to an embodiment of the disclosure, the processor may extracta Y component that is a brightness component of each unit of the viewinformation. According to an embodiment of the disclosure, the processormay multiplex Y components of respective units to a Y channel that is abrightness channel of a view.

In operation S1222, the processor may perform Fourier transform on thebrightness component or the brightness channel. The processor acquires aFourier spectrum of the brightness component or the brightness channelthrough the Fourier transform. According to an embodiment of thedisclosure, the processor may use a Gaussian window to smooth a boundarypart of the spectrum.

In operation S1223, the processor performs a phase congruency analysison the acquired Fourier spectrum. The processor searches the Fourierspectrum for information regarding a detail through the phase congruencyanalysis. According to an embodiment of the disclosure, the processormay search for the information regarding a detail through an operationof separating complex vectors orienting to a specific direction in theFourier spectrum.

In operation S1224, the processor performs a rebalance spectrumoperation based on the retrieved information regarding a detail. Therebalance spectrum operation is an operation of enhancing a retrieveddetail. According to an embodiment of the disclosure, the processor mayenhance the detail through an operation of increasing a magnitude of alength of retrieved complex vectors, that is, a magnitude of an absolutevalue.

In operation S1225, the processor performs Fourier inverse transform onthe brightness component or the brightness channel on which therebalance spectrum operation has been completed. The processor acquiresan enhanced new brightness component or brightness channel Y′ throughthe Fourier inverse transform.

In operation S1226, enhanced information is output. According to anembodiment of the disclosure, the processor may combine informationregarding all units on which the processing has been performed, by usinga Gaussian window such that overlapping is smoothly processed.

The processor combines the new brightness channel Y′ and the initialcolor channels U and V to a color space model Y′UV. The color spacemodel Y′UV is converted into a color space model RGB, and accordingly,enhanced view information of a scene may be acquired using the colorspace model RGB.

FIG. 14 is a flowchart of a method of displaying a scene according toanother embodiment of the disclosure.

Referring to FIG. 14, a display system may perform anti-aliasingprocessing based on adjustment information. The display system mayprovide a relatively clear and realistic experience to a user by usingan anti-aliasing processing operation to prevent a pixel staircasephenomenon and edge distortion.

In operation S1410, a processor receives a plurality of pieces of viewinformation corresponding to a scene. According to an embodiment of thedisclosure, the scene may be a real scene or a virtual scene. Theplurality of pieces of view information may be an intrinsic imageacquired from a plurality of matched cameras having differentviewpoints. Alternatively, the processor may form a plurality of piecesof view information corresponding to a virtual scene by itself by usinga rendering program.

In operation S1420, the processor may acquire adjustment informationfrom the plurality of pieces of view information. According to anembodiment of the disclosure, the processor may generate a view matrixthat is a matrix including each view information included in theplurality of pieces of view information, based on a geometric parameter.The view matrix is a matrix representing a light field of acorresponding scene to be displayed.

In operation S1430, the processor may perform the anti-aliasingprocessing operation based on the adjustment information.

According to an embodiment of the disclosure, the processor mayfactorize the view matrix included in the adjustment information to aproduct of a matrix indicating light intensity of a display included ina mobile electronic device and a matrix indicating transparency of aspatial light modulator.

According to an embodiment of the disclosure, the processor may performthe factorization by using a WRRI algorithm. In more detail, asdescribed with reference to Equation 9, the processor may calculateoptimized intensity of the display and optimized transparency of thespatial light modulator through an Hadamard product with respect to agiven light field by using the WRRI algorithm.

The processor may perform the anti-aliasing processing operation on theview matrix. According to an embodiment of the disclosure, the processormay perform the anti-aliasing processing operation by using pixelbarycentric coordinates. According to an embodiment of the disclosure,the processor may perform the anti-aliasing processing operation in anoperation of performing light field factorization on the view matrix.For example, the processor may perform the anti-aliasing processingoperation by calculating and using a weighted matrix in the operation ofperforming light field factorization.

In operation S1440, the processor may control transparency of a spatiallight modulator and light intensity of a display based on the adjustmentinformation. According to an embodiment of the disclosure, the processormay factorize the view matrix to a column vector and a row vector,control the transparency of the spatial light modulator based on thecolumn vector, and control the light intensity of the display based onthe row vector. According to an embodiment of the disclosure,transparency of each pixel of the spatial light modulator may beadjusted in response to a transparency control signal provided from theprocessor, and light intensity of each pixel of the display may beadjusted in response to an intensity control signal provided from theprocessor.

The display system may display a scene to the user based on theoperations described above. According to an embodiment of thedisclosure, light emitted from the display based on the adjusted lightintensity is delivered to the user through an optical lens by passingthrough the spatial light modulator having the adjusted transparency.The display system may provide a light field which provides anexperience approximate to a 3D effect in the real to the user byproviding light concentrated through the optical lens to the user.

FIG. 15 is a flowchart of a method of displaying a scene according to anembodiment of the disclosure.

Referring to FIG. 15, a display system may perform an enhancingprocessing operation and an anti-aliasing processing operation on piecesof view information. The display system may provide a relatively clearand realistic experience to a user by using the enhancing processingoperation and the anti-aliasing processing operation to enhance a detailof views and prevent a pixel staircase phenomenon and edge distortion.

In operation S1510, a processor receives a plurality of pieces of viewinformation corresponding to a scene. According to an embodiment of thedisclosure, the scene may be a real scene or a virtual scene.

In operation S1520, the processor performs the enhancing processingoperation on the pieces of view information. The enhancing processingoperation is an operation of enhancing only a detail while maintainingcolor information of the pieces of view information as it is. Accordingto an embodiment of the disclosure, for the enhancing processingoperation, the processor may separate only a brightness channel fromeach view information and perform a processing operation on thebrightness channel. According to an embodiment of the disclosure, theprocessor may use Fourier transform and phase congruency analysis forthe enhancing processing operation.

In operation S1530, the processor may acquire adjustment informationfrom the plurality of pieces of view information. According to anembodiment of the disclosure, the processor may generate a view matrixthat is a matrix including each view information included in theplurality of pieces of view information, based on a geometric parameter.The view matrix is a matrix representing a light field of acorresponding scene to be displayed.

In operation S1540, the processor may perform the anti-aliasingprocessing operation based on the adjustment information.

According to an embodiment of the disclosure, the processor mayfactorize the view matrix included in the adjustment information to aproduct of a matrix indicating light intensity of a display included ina mobile electronic device and a matrix indicating transparency of aspatial light modulator.

According to an embodiment of the disclosure, the processor may performthe factorization by using a WRRI algorithm. In more detail, asdescribed with reference to Equation 9, the processor may calculateoptimized intensity of the display and optimized transparency of thespatial light modulator through an Hadamard product with respect to agiven light field by using the WRRI algorithm.

The processor may perform the anti-aliasing processing operation on theview matrix. According to an embodiment of the disclosure, the processormay perform the anti-aliasing processing operation by using pixelbarycentric coordinates. According to an embodiment of the disclosure,the processor may perform the anti-aliasing processing operation in anoperation of performing light field factorization on the view matrix.For example, the processor may perform the anti-aliasing processingoperation by calculating and using a weighted matrix in the operation ofperforming light field factorization.

In operation S1550, the processor may control transparency of a spatiallight modulator and light intensity of a display based on the adjustmentinformation. According to an embodiment of the disclosure, the processormay factorize the view matrix to a column vector and a row vector,control the transparency of the spatial light modulator based on thecolumn vector, and control the light intensity of the display based onthe row vector. According to an embodiment of the disclosure,transparency of each pixel of the spatial light modulator may beadjusted in response to a transparency control signal provided from theprocessor, and light intensity of each pixel of the display may beadjusted in response to an intensity control signal provided from theprocessor.

The display system may display a scene to the user based on theoperations described above. According to an embodiment of thedisclosure, light emitted from the display based on the adjusted lightintensity is delivered to the user through an optical lens by passingthrough the spatial light modulator having the adjusted transparency.The display system may provide a light field which provides anexperience approximate to a 3D effect in the real to the user byproviding light concentrated through the optical lens to the user.

The technical features of the disclosure could be clearly described withreference to the above-described embodiments and the accompanyingdrawings. It would be obvious to those of ordinary skill in the art thatthe technical features of the disclosure could also be modified andimplemented by other embodiments without departing from the technicalfeatures of the disclosure. Therefore, the embodiments disclosed in thespecification and the accompanying drawings should be understood in theillustrative sense only and not for the purpose of limitation. Acomponent expressed in a singular form does not exclude the featurewherein the component exists plural in number unless they are defineddifferently.

The disclosed embodiments may be implemented in a form of anon-transitory computer-readable recording medium configured to storecomputer-executable instructions and data. The instructions may bestored in a form of program codes and may perform, when executed by aprocessor, a certain operation by generating a certain program module.In addition, the instructions may perform certain operations of thedisclosed embodiments when executed by the processor.

Certain aspects of the disclosure can also be embodied as computerreadable code on a non-transitory computer readable recording medium. Anon-transitory computer readable recording medium is any data storagedevice that can store data which can be thereafter read by a computersystem. Examples of the non-transitory computer readable recordingmedium include a Read-Only Memory (ROM), a Random-Access Memory (RAM),Compact Disc-ROMs (CD-ROMs), magnetic tapes, floppy disks, and opticaldata storage devices. The non-transitory computer readable recordingmedium can also be distributed over network coupled computer systems sothat the computer readable code is stored and executed in a distributedfashion. In addition, functional programs, code, and code segments foraccomplishing the disclosure can be easily construed by programmersskilled in the art to which the disclosure pertains.

At this point it should be noted that the various embodiments of thedisclosure as described above typically involve the processing of inputdata and the generation of output data to some extent. This input dataprocessing and output data generation may be implemented in hardware orsoftware in combination with hardware. For example, specific electroniccomponents may be employed in a mobile device or similar or relatedcircuitry for implementing the functions associated with the variousembodiments of the disclosure as described above. Alternatively, one ormore processors operating in accordance with stored instructions mayimplement the functions associated with the various embodiments of thedisclosure as described above. If such is the case, it is within thescope of the disclosure that such instructions may be stored on one ormore non-transitory processor readable mediums. Examples of theprocessor readable mediums include a ROM, a RAM, CD-ROMs, magnetictapes, floppy disks, and optical data storage devices. The processorreadable mediums can also be distributed over network coupled computersystems so that the instructions are stored and executed in adistributed fashion. In addition, functional computer programs,instructions, and instruction segments for accomplishing the disclosurecan be easily construed by programmers skilled in the art to which thedisclosure pertains.

It should be understood that embodiments described herein should beconsidered in a descriptive sense only and not for purposes oflimitation. Descriptions of features or aspects within each embodimentshould typically be considered as available for other similar featuresor aspects in other embodiments.

While the disclosure has been shown and described with reference tovarious embodiments thereof, it will be understood by those skilled inthe art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims and their equivalents.

What is claimed is:
 1. A system for displaying an image in units of ascene, the system comprising: a display configured to emit light; aspatial light modulator configured to modulate input light based on atransparency value; and at least one processor configured to: acquireadjustment information comprising transparency of the spatial lightmodulator and light intensity of the display from a plurality of piecesof view information corresponding to the scene, and adjust an intensityvalue of the light emitted from the display and the transparency valueof the spatial light modulator based on the adjustment information,wherein the plurality of pieces of view information are opticalinformation of the scene, the optical information having been acquiredat a plurality of viewpoints.
 2. The system of claim 1, wherein theadjustment information comprises a view matrix factorized to a rowvector and a column vector, and wherein the at least one processor isfurther configured to: control the intensity value of the light based onthe row vector, and control the transparency value of the spatial lightmodulator based on the column vector.
 3. The system of claim 2, whereinthe at least one processor is further configured to factorize the viewmatrix to the row vector and the column vector by a weighted rank-1residue iteration (WRRI) method.
 4. The system of claim 3, wherein theat least one processor is further configured to: generate a weightedmatrix based on barycentric coordinates of the view matrix, andfactorize the view matrix, on which anti-aliasing processing has beenperformed by using the weighted matrix, by the WRRI method.
 5. Thesystem of claim 2, wherein the at least one processor is furtherconfigured to form the view matrix based on a geometric parameter, andwherein the geometric parameter comprises a distance between the displayand the spatial light modulator.
 6. The system of claim 2, wherein theat least one processor is further configured to: perform an enhancingprocessing operation on the plurality of pieces of view information byextracting a brightness channel of the plurality of pieces of viewinformation, and acquire the view matrix from the enhancing-processedplurality of pieces of view information.
 7. The system of claim 6,wherein the at least one processor is further configured to: segment theplurality of pieces of view information into a plurality of units whichoverlap each other, extract first brightness components from theplurality of units, generate second brightness components by retrievingand enhancing detail vectors based on the first brightness components,and perform the enhancing processing operation by using the secondbrightness components.
 8. The system of claim 7, wherein the at leastone processor is further configured to retrieve the detail vectors byusing phase congruency analysis.
 9. The system of claim 8, wherein theat least one processor is further configured to enhance the detailvectors by using a rebalance spectrum operation.
 10. The system of claim2, further comprising: a mobile electronic device comprising the displayand the at least one processor; and a mounting device configured tomount the mobile electronic device to a head of a user, wherein themounting device is further configured to accommodate the mobileelectronic device and the spatial light modulator therein.
 11. Thesystem of claim 10, wherein the mounting device comprises at least oneof a virtual reality (VR) helmet or VR glasses.
 12. The system of claim10, wherein the plurality of pieces of view information are acquiredusing a plenoptic camera and stored in a memory of the mobile electronicdevice, and wherein the at least one processor is further configured toaccess the memory to extract the plurality of pieces of viewinformation.
 13. The system of claim 1, wherein the spatial lightmodulator comprises a liquid crystal layer and at least one polarizingplate.
 14. The system of claim 2, wherein the at least one processor isfurther configured to acquire the plurality of pieces of viewinformation by using a rendering program.
 15. A scene display method ofdisplaying an image in units of a scene, the method comprising:receiving a plurality of pieces of view information corresponding to thescene; acquiring, from the plurality of pieces of view information,adjustment information comprising intensity of light emitted from adisplay and transparency of a spatial light modulator configured tomodulate the light; and adjusting an intensity value of the lightemitted from the display and a transparency value of the spatial lightmodulator, based on the adjustment information, wherein the plurality ofpieces of view information are optical information of the scene, theoptical information having been acquired at a plurality of viewpoints.16. The method of claim 15, wherein the acquiring of the adjustmentinformation comprises acquiring a view matrix to be factorized to a rowvector and a column vector, from the plurality of pieces of viewinformation corresponding to the scene, and wherein the adjusting of theintensity value of the light emitted from the display and thetransparency value of the spatial light modulator, based on theadjustment information, comprises controlling the intensity value of thelight based on the row vector and controlling the transparency value ofthe spatial light modulator based on the column vector.
 17. The methodof claim 16, wherein the factorizing of the view matrix comprisesfactorizing the view matrix by a weighted rank-1 residue iteration(WRRI) method, and wherein the factorizing of the view matrix by theWRRI method comprises generating a weighted matrix based on barycentriccoordinates of the view matrix and factorizing the view matrix, on whichanti-aliasing processing has been performed by using the weightedmatrix, by the WRRI method.
 18. The method of claim 15, furthercomprising performing an enhancing processing operation on the pluralityof pieces of view information by extracting a brightness channel of theplurality of pieces of view information, wherein the acquiring of theview matrix from the plurality of pieces of view information comprisesacquiring the view matrix from the enhancing-processed plurality ofpieces of view information.
 19. The method of claim 18, wherein theperforming of the enhancing processing operation comprises: segmentingthe plurality of pieces of view information into a plurality of unitswhich overlap each other; extracting first brightness components fromthe plurality of units; generating second brightness components byretrieving and enhancing detail vectors based on the first brightnesscomponents; and performing the enhancing processing operation by usingthe second brightness components.
 20. At least one non-transitorycomputer-readable computer program product having stored thereincomputer program codes, which when read and executed by at least oneprocessor, perform a scene display method of displaying an image inunits of a scene, the least one non-transitory computer-readablecomputer program product comprising a recording medium having storedtherein a program configured to perform: emitting light from a display;modulating the light based on a transparency value; acquiring, from aplurality of pieces of view information corresponding to the scene,adjustment information comprising transparency of a spatial lightmodulator and light intensity of the display; and adjusting an intensityvalue of the light emitted from the display and a transparency value ofthe spatial light modulator, based on the adjustment information,wherein the plurality of pieces of view information are opticalinformation of the scene, the optical information having been acquiredat a plurality of viewpoints.